Skip to content

Fix incorrect buffer slicing logic in VarChar and VarBinary Accessors - #102

Open
programmer314 wants to merge 1 commit into
dremio:masterfrom
programmer314:bugfix/accessor-getstream-calc-buffer-slice-len-correctly
Open

Fix incorrect buffer slicing logic in VarChar and VarBinary Accessors#102
programmer314 wants to merge 1 commit into
dremio:masterfrom
programmer314:bugfix/accessor-getstream-calc-buffer-slice-len-correctly

Conversation

@programmer314

Copy link
Copy Markdown

Description

This PR addresses a functional bug in how variable-length data is sliced from Arrow buffers within the SqlAccessor implementations.

Previously, the getStream(int index) methods in VarCharAccessor and VarBinaryAccessor were passing the absolute end offset (h.end) as the length parameter to the buffer slice method. In the Netty/Arrow API, the signature is slice(int index, int length).

Using the end offset as the length caused the resulting stream to "bleed," including not only the target record but also all subsequent data remaining in the vector's buffer.

Changes

  • VarCharAccessor.java: Updated getStream to calculate length as h.end - h.start.
  • VarBinaryAccessor.java: Updated getStream to calculate length as h.end - h.start.
  • Consistency: Standardized the slicing logic across both variable-length accessors to ensure getReader() and getStream() return the correct byte ranges.

Technical Breakdown

When fetching a record from a variable-width vector:

  1. h.start: The byte offset where the value begins.
  2. h.end: The byte offset where the value ends.
  3. The Bug: slice(h.start, h.end) incorrectly treats the end-pointer as the count of bytes to read starting from h.start.
  4. The Fix: slice(h.start, h.end - h.start) ensures the slice is bounded strictly to the bytes belonging to that specific record.

Impact

  • Data Integrity: Prevents getReader() and getStream() from returning concatenated data from subsequent rows.
  • Reliability: Ensures that downstream consumers of these streams do not process "dirty" data or encounter unexpected characters at the end of a string.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant